Sequential-Access FM-Indexes

Previous authors have shown how to build FM-indexes efficiently in external memory, but querying them efficiently remains an open problem. Searching näıvely for a pattern P requires Θ(|P |) random access. In this paper we show how, by storing a few small auxiliary tables, we can access data only in the order in which they appear on disk, which should be faster. An FM-index [4] is a compressed representation of a text that allows us to quickly search for arbitrary patterns in that text. Their growing popularity in genomics (e.g., in BWT-SW, Bowtie, SOAP2 and BWA) means we should look for ways in which they can handle massive datasets, which may have to reside in external memory even when compressed. Unfortunately, although we know how to build FM-indexes efficiently in external memory [3], querying them efficiently remains an open problem. Searching näıvely for a pattern P requires Θ(|P |) random access, which are expensive due to seek times. We refer the reader to the papers by Chien et al. [1], Hon et al. [5] and Ferragina [2] for more discussion of this problem. In this paper we extend a result by Orlandi and Venturini [6] to show how, by storing a few small auxiliary tables, we can access data only in the order in which they appear on disk. We may read slightly more data but, since sequential access to disk is orders of magnitude faster than random access, our modified index should be faster overall. FM-indexes are based on the Burrows-Wheeler Transform (BWT), which permutes the characters of a string T based on the contexts that follow them. We can compute B = BWT(T ) by lexicographically sorting the rotations of T , then recording the last character of each rotation. (If we want to recover T later, we append a special symbol before computing B or record the position to which a designated character is mapped.) For example, if T = 110111100101110101010001111 , then B = 110111011001001011111010110 . We use binary strings for simplicity but the results in this paper extend to any reasonable alphabet size. Notice that, for any pattern P , the characters immediately preceding occurrences of P in T are adjacent in B (considering T to be cyclic). For example, if P = 0101 then the characters immediately preceding occurrences of T are T [8], T [14] and T [16], which are mapped to B[7], B[6] and B[5], respectively. We call B[5..7] the interval for P = 0101. The basic operation of FM-indexes is to find the interval in B for any given pattern P . For example, the length of the interval is the number of occurrences of P in T . Notice that the left endpoint of the interval is the rank of the lexicographically first rotation of T that starts with P , and the right endpoint is the rank of the lexicographically last such rotation. To find these endpoints, we store data structures such that, for any character c in the alphabet and any position i in B, we can quickly compute the number rankc(i) of occurrences of c in B[1..i]. We also store the number C[c] of characters in B lexicographically less than c. Suppose we are näıvely searching for the right endpoint of the interval; finding the left endpoint is essentially symmetric. We iteratively compute j1 = rankP [|P |](|B|) + C [